Display status modifying apparatus and method, display status modifying program and storage medium storing the same, picture providing apparatus and method, picture providing program and storage medium storing the same, and picture providing system

ABSTRACT

In a display status modifying apparatus, objects each including a temporally continuous picture of an arbitrary shape are received from the objects are displayed on a predetermined display according to display specification information that specifies display status of the objects, and specified content of the display specification information is modified according to a modification instruction for modifying the display status of the objects displayed on the display unit, input via an input unit. Accordingly, intention of a user can be reflected, serving to popularize distribution of picture.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a display status modifying apparatus, adisplay status modifying method, a display status modifying program, astorage medium storing the same, a picture providing apparatus, apicture providing method, a picture providing program, a storage mediumstoring the same, and a picture providing system. The present inventionmay be suitably applied, for example, to a picture providing systemconstructed of a motion picture distribution server and a personalcomputer linked with each other via the Internet.

2. Description of the Related Art

Recently, picture providing systems, in which motion picture data beingacquired by imaging a target with a video camera, etc. is distributedfrom a motion picture distribution server to a personal computer via theInternet so that motion picture based on the motion picture data isdisplayed in real time on a display unit of the personal computer, arebecoming commonly available.

According to the picture providing systems, however, motion picture datais unilaterally distributed to the personal computer from the motionpicture distribution server and motion picture based on the motionpicture data is inflexibly displayed on the display unit of the personalcomputer. Thus, the picture providing systems have not necessarily beensuccessful in providing motion picture as desired by the user, and thushave not been sufficient to popularize distribution of motion picture.

SUMMARY OF THE INVENTION

The present invention has been made in view of the above, and an objectof thereof is to provide a display status modifying apparatus, a displaystatus modifying method, a display status modifying program, a storagemedium storing the same, a picture providing apparatus, a pictureproviding method, a picture providing program, a storage medium storingthe same, and a picture providing system that serve to popularizedistribution of picture.

To this end, according to one aspect of the present invention, an objectincluding a temporally continuous picture of an arbitrary shape isreceived from outside, the object is displayed on a predetermineddisplay unit according to display specification information thatspecifies a display status of the object, and specified content of thedisplay specification information is modified according to amodification instruction for modifying the display status of the objectdisplayed on the display unit, input via an input unit. Accordingly,intention of a user can be reflected, serving to popularize distributionof picture.

According to another aspect of the present invention, in a pictureproviding apparatus, an object including a temporally continuous pictureof an arbitrary shape is generated, display specification informationthat specifies a display status of the object and in which modificationof specified content of the display status is allowed is generated, andthe object and the display specification information are transmitted; ina display status modifying apparatus, the object and the displayspecification information are received, the object is displayed on apredetermined display unit according to the display specificationinformation, and specified content of the display specificationinformation is modified according to a modification instruction formodifying the display status of the object displayed on the displayunit, input via an input unit. Accordingly, a picture in which intentionof a user is reflected can be provided, serving to popularizedistribution of picture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the overall construction of a motionpicture distribution system according to a first embodiment of thepresent invention;

FIG. 2 is a block diagram showing the construction of a motion picturedistribution server according to the first embodiment;

FIG. 3 is a schematic diagram showing a first example of tracking areaspecification;

FIG. 4 is a schematic diagram showing a first example of objectextraction;

FIG. 5 is a schematic diagram showing the configuration of a maskpicture for specifying the shape of an object;

FIG. 6 is a flowchart showing the procedure of a motion picturedistribution process;

FIG. 7 is a block diagram showing the construction of a personalcomputer according to the first embodiment;

FIG. 8 is a schematic diagram showing an example of window screenaccording to the first embodiment;

FIG. 9 is a schematic diagram showing a first example of display ofobjects in a display area;

FIG. 10 is a schematic diagram showing a first example of modificationof display configuration;

FIG. 11 is a schematic diagram showing a second example of modificationof display configuration;

FIG. 12 is a schematic diagram showing a third example of modificationof display configuration;

FIG. 13 is a schematic diagram showing a fourth example of modificationof display configuration;

FIG. 14 is a schematic diagram showing a fifth example of modificationof display configuration;

FIG. 15 is a schematic diagram showing a sixth example of modificationof display configuration;

FIG. 16 is a flowchart showing the procedure of a display process (A);

FIG. 17 is a flowchart showing the procedure of a display process (B);

FIG. 18 is a flowchart showing the procedure of an erasing process;

FIG. 19 is a flowchart showing the procedure of an erasure cancellingprocess;

FIG. 20 is a flowchart showing the procedure of an enlarging/reducingprocess;

FIG. 21 is a flowchart showing the procedure of a layer changingprocess;

FIG. 22 is a block diagram showing the overall construction of a motionpicture distribution system according to a second embodiment of thepresent invention;

FIG. 23 is a block diagram showing the construction of an arbitraryshape encoding apparatus according to the second embodiment;

FIGS. 24A and 24B are schematic diagrams showing a second example oftracking area specification;

FIG. 25 is a block diagram showing the construction of a personalcomputer according to the second embodiment;

FIG. 26 is a schematic diagram showing an example of window screenaccording to the second embodiment;

FIG. 27 is a schematic diagram showing a second example of display ofobjects in a display area;

FIG. 28 is a schematic diagram showing a seventh example of modificationof display configuration;

FIG. 29 is a schematic diagram showing an eighth example of modificationof display configuration;

FIG. 30 is a schematic diagram showing a ninth example of modificationof display configuration;

FIG. 31 is a schematic diagram showing a tenth example of modificationof display configuration;

FIG. 32 is a schematic diagram showing a third example of tracking areaspecification;

FIG. 33 is a schematic diagram showing an eleventh example ofmodification of display configuration;

FIG. 34 is a flowchart showing the procedure of a display process (C);

FIG. 35 is a flowchart showing the procedure of a display process (D);and

FIG. 36 is a schematic diagram showing a second example of objectextraction.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail with reference to the accompanying drawings.

1. First Embodiment

1.1 Overall Construction

Referring to FIG. 1, a motion picture providing system 1 according to afirst embodiment of the present invention is constructed by linking amotion picture distribution server 2 on the providing side and apersonal computer 3 via the Internet 4.

The motion picture distribution server 2 tracks a predetermined area(hereinafter referred to as a tracking area) in frame pictures based onframe data constituting motion picture data to be distributed, andcompression encodes each unit of extracted picture data of extractedpictures extracted from the tracking area, distributing the encoded datato the personal computer 3.

The personal computer 3 is arranged such that, each time encoded data isdistributed from the motion picture distribution server 2, objects basedon associated extracted picture data can be sequentially displayedthereon.

1.2 Construction of Motion Picture Distribution Server

Referring to FIG. 2, when a motion vector detection unit 7 of the motionpicture distribution server 2 receives motion picture distributionrequest data Dre transmitted from the personal computer 3 via adistribution-side transmission/reception unit 12, the motion vectordetection unit 7 reads frame data (hereinafter referred to as subjectframe data) D1 to be encoded from motion picture data MD stored in anHDD 6, and divides it into a plurality of macro blocks, generatingdivided subject frame data (hereinafter referred to as divided framedata).

The motion vector detection unit 7 reads reference frame data that is,for example, temporally previous relative to the divided frame data D2,and detects motion vector data D3 of each of the macro blocks by blockmatching of the divided frame data D2 and the reference frame data,forwarding the motion vector data D3 and the divided frame data D2 toeach of object processing units CP1, CP2, and CP3.

A motion tracking unit 8A of the object processing unit CP1 is adaptedto receive input of tracking specification data D15 specifying arectangular tracking area including, for example, a person to be trackedin the frame picture based on the divided frame data D2.

In this embodiment, in the motion tracking unit 8A, for example, asshown in FIG. 3, a tracking area (hereinafter referred to as a scenerytracking area) 32 including a scenery picture (hereinafter referred toas a scenery object) 31 in the proximity of the top left corner of aframe picture 30 based on frame data is specified in advance by thetracking specification data D15.

Thus, out of the motion vector data D3 forwarded from the motion vectordetection unit 7, the motion tracking unit 8A (FIG. 2) uses the motionvector data of all the macro blocks associated with the scenery trackingarea 32 to roughly detect the movement of each of the macro blocksassociated with the scenery tracking area 32 between each adjacent pairof divided frame data D2 forwarded from the motion vector detection unit7.

Furthermore, the motion tracking unit 8A detects the scenery trackingarea 32, having been roughly detected, now more specifically by linematching between each adjacent pair of divided frame data D2 so that themovement of each of the macro blocks associated with the scenerytracking area 32 is detected with high precision.

Furthermore, the motion tracking unit 8A separates the scenery trackingarea 32 specified by the tracking specification data D15, as shown inFIG. 4, forwarding it to an arbitrary shape extraction unit 9A asscenery tracking data D4 (FIG. 2).

The arbitrary shape extraction unit 9A is adapted to accept fromoutside, when the tracking specification data D15 is input to the motiontracking unit 8A, input of extraction specification data D18 specifyingan arbitrary point on an object, whereby an object to be extracted fromthe tracking area specified by the tracking specification data D15 isspecified.

In this embodiment, as shown in FIG. 3, in the arbitrary shapeextraction unit 9A, an arbitrary point 31A that specifies the sceneryobject 31 is specified bay the extraction specification data D18.

Thus, as shown in FIG. 4, the arbitrary shape extraction unit 9A detectsthe edge of the scenery object 31 from the scenery tracking area 32based on the scenery tracking data D4 forwarded from the motion trackingunit 8A, with an assumption that the center of the scenery object 31 isa pixel associated with the point 31A specified by the extractionspecification data D18, leaving the original picture as it is inside theedge while replacing the outside with, for example, pixels representingblack so that it will not be displayed on the reception side, wherebythe scenery object 31 is extracted.

Furthermore, the arbitrary shape extraction unit 9A generates a maskpattern picture 37 for exposing the extracted scenery object 31 whilecovering the area other surrounding the scenery object 31 so that theshape of the scenery object 31 will be presented to the reception side,as shown in FIG. 5.

The arbitrary shape extraction unit 9A forwards data of the extractedscenery object 31 (in this case including the surrounding area in thescenery tracking area 32, replaced with black pixels) and data of themask pattern picture 37 (hereinafter referred to as mask pattern data)to an arbitrary shape encoding unit 10A as scenery object data D5 (FIG.2).

The arbitrary shape encoding unit 10A compresses and encodes the objectdata in the scenery object data D5, by the unit of macro block of thesame size as that in the original picture, by a relatively lowcompression rate based on MPEG4 compression encoding, generating encodeddata, and also compresses and encodes the mask pattern data in thescenery object data D5 by a relatively low compression rate, generatingencoded mask pattern data.

The arbitrary shape encoding unit 10A forwards the encoded data andencoded mask pattern data to a multiplexing unit 11 as encoded scenerydata D6.

As described above, each time the divided frame data D2 is input fromthe motion vector detection unit 7, the object processing unit CP1sequentially detects the movement of each of the macro blocks associatedwith the scenery tracking area 32 specified by the trackingspecification data D15 from the frame pictures based on the dividedframe data D2, thereby separating the scenery tracking area 32.

Upon each time separating the scenery tracking area 32, the objectprocessing unit CP1 sequentially compresses and encodes, according toMPEG4, the scenery object data D5 of the scenery object 31 extractedfrom the scenery tracking area 32 based on the extraction specificationdata D18, generating the encoded scenery data D6, and forwards it to themultiplexing unit 11.

Similarly to the motion tracking unit 8A, motion tracking units 8B and8C are adapted to accept input of tracking specification data D16 andD17, respectively, each specifying a rectangular tracking areaincluding, for example, a person to be tracked in the frame picturesbased on the divided frame data D2. Also, similarly to the arbitraryshape extraction unit 9A, arbitrary shape, extraction units 9B and 9Care adapted to accept from outside, when the tracking specification dataD16 and D17 are input to the motion tracking units 8B and 8C, input ofextraction specification data D19 and D20, respectively, each specifyingan arbitrary point on an object, whereby objects to be extracted fromthe tracking areas specified by the tracking specification data D16 andD17 are specified.

In this embodiment, as shown in FIG. 3, in the motion tracking unit 8B,a tracking area (hereinafter referred to as a telop tracking area) 34including a telop (hereinafter referred to as a telop object) 33 in theproximity of the top right corner of the frame picture 30 based on theframe data is specified in advance by the tracking specification dataD16. In the motion tracking unit 8C, a tracking area (hereinafterreferred to as a person tracking area) 36 including a picture of aperson (hereinafter referred to as a person object) 35 in the proximityof the bottom right corner of the frame picture 30 based on the framedata is specified in advance by the tracking specification data D17.

Furthermore, in the arbitrary shape extraction unit 9B, an arbitrarypoint 33A that specifies the telop object 33 is specified by theextraction specification data D19. In the arbitrary shape extractionunit 9C, an arbitrary point 35A that specifies the person object 35 isspecified by the extraction specification data D20.

Thus, similarly to the object processing unit CP1, each time the dividedframe data D2 is input from the motion vector detection unit 7, theobject processing units CP2 and CP3 (FIG. 2) detect the movement of eachof the macro blocks associated with the telop tracking area 34 and theperson tracking area 36 specified by the tracking specification data D16and D17 from the frame pictures based on the divided frame data D2,separating the telop tracking area 34 and the person tracking area 36,respectively (telop tracking data D7 and person tracking data D10).

Upon each time separating the telop tracking area 34 and the persontracking area 36, the object processing units CP2 and CP3 sequentiallycompress and encode, according to MPEG4, telop object data D8 of thetelop object 33 and person object data D11 of the person object 35extracted from the telop tracking area 34 and the person tracking area36 based on the extraction specification data D19 and D20, respectively,generating encoded telop data D9 and encoded person data D12, andforwards them to the multiplexing unit 11.

A display specification information generation unit 13 calculates acenter coordinate point and a reference point of the scenery object 31based on the scenery object data D5 forwarded from the arbitrary shapeextraction unit 9A, a center coordinate point and a reference point ofthe telop object 33 based on the telop object data D8 forwarded from thearbitrary shape extraction unit 9B, and a center coordinate point and areference point of the person object 35 based on the person object dataD11 forwarded from the arbitrary shape extraction unit 9C, for eachdivided frame data D2 of, for example, 60 frames (hereinafter referredto as a frame data set).

The display specification information generation unit 13 alsocalculates, for each frame data set, a picture size of the sceneryobject 31, a picture size of the telop object 33, and a picture size ofthe person object 35 based on the object data D5, D8, and D11 forwardedfrom the arbitrary shape extraction units, 9A, 9B, and 9C, respectively.

The display specification information generation unit 13 is adapted toaccept from outside, when the tracking specification data D15, D16, andD17 are input to the motion tracking units 8A, 8B, and 8C, respectively,input of display specification data D21 specifying whether or not todisplay each of the object and layer order specification data D22specifying an order of displaying the objects overlappingly (hereinafterreferred to as layer order).

In this embodiment, in the display specification information generationunit 13, the scenery object 31, the telop object 33, and the personobject 35 are specified for display by the display specification dataD21, and the scenery object 31 is specified to be on the uppermostlayer, the telop object 33 on the intermediate layer, and the personobject 35 on the lowermost layer by the layer order specification dataD22.

The display specification information generation unit 13 forwards thecenter coordinate point and the reference point of each of the objectsand the picture size of each of the objects, calculated for each framedata set, and the display specification data D21 and the layer orderspecification data D22, supplied from outside, to a displayspecification information encoding unit 14 as display specificationinformation D25 for specifying display status of each of the objects.

The display specification information generation unit 13 generates thedisplay specification information D25 on the basis of each frame dataset, not on the basis of each divided frame data D2, so that the load ofthe encoding process by the display specification information encodingunit 14, to which the display specification information D25 isforwarded, will be minimized.

Upon each time receiving the display specification information D25forwarded from the display specification information generation unit 13,the display specification information encoding unit 14 sequentiallycompresses and encodes the display specification information D25,generating encoded display specification data D26, and forwards it tothe multiplexing unit 11.

Upon each time receiving the encoded display specification data D26forwarded from the display specification information generation unit 13,and the encoded scenery data D6, the encoded telop data D9, and theencoded person data D12 forwarded from the arbitrary shape encodingunits 10A, 10B, and 10C, respectively, the multiplexing unit 11multiplexes them as required to generate multiplexed data D13, which isdistributed to the personal computer 3 via the distribution sidetransmission/reception unit 12.

1.3 Procedure of Providing Picture

The procedure for providing a picture from the motion picturedistribution server 2 to the personal computer 3 will be summarizedbelow. Referring to FIG. 6, upon starting a routine RT1, the motionpicture distribution server 2 enters step SP1.

In step SP1, the motion picture distribution server 2 determines whethera request for distributing a motion picture has been received from thepersonal computer 3. If the test turns out negative, indicating thatmotion picture distribution request data Dre has not been received fromthe personal computer 3, the motion picture distribution server 2 waitsuntil a motion picture distribution request is received from thepersonal computer 3.

If the test turns out positive, indicating that motion picturedistribution request data Dre has been received from the personalcomputer 3, the motion picture distribution server 2 proceeds to stepSP2.

In step SP2, the motion picture distribution server 2 detects motionvector data D2 of each of the macro blocks by block matching, and thenproceeds to step SP3.

In step SP3, the motion picture distribution server 2 detects themovement of each of the macro blocks associated with the tracking areas32, 34, and 36 specified by tracking, specification data D15 to D17,respectively, and then proceeds to step SP4.

In step SP4, the motion picture distribution server 2 separates thetracking areas 32, 34, and 36 specified by the tracking specificationdata D15 to D17 based on the detected movement of each of the macroblocks associated with the tracking areas 32, 34, and 36, respectively,and then proceeds to step SP5.

In step SP5, the motion picture distribution server 2 extracts theobjects 31, 33, and 35 from the separated tracking areas 32, 34, and 36based on the extraction specification data D18 to D20, respectively, andthen proceeds to steps SP6 and SP7.

In step SP6, the motion picture distribution server 2 compresses andencodes the object data D5, D8, and D11 of the objects 31, 33, and 35according to MPEG4, generating encoded data D6, D9, and D12, and thenproceeds to step SP9.

In step SP7, the motion picture distribution server 2 calculates acenter coordinate point, a reference point, and a picture size of eachof the objects 31, 33, and 35 for each frame data set, generatingdisplay specification information D25 including the center coordinatepoint and the reference point of each of the objects, the picture sizeof each of the objects, the display specification data D21, and thelayer order specification data D22, and then proceeds to step SP8.

In step SP8, the motion picture distribution server 2 compresses andencodes the display specification information D25 according to MPEG4,generating encoded display specification data D26, and then proceeds tostep SP9.

In step SP9, the motion picture distribution server 2 multiplexes thecompression encoded data D6, D9, D12 and the encoded displayspecification data D26 as required to generate multiplexed data D13, andthen proceeds to step SP10.

In step SP10, the motion picture distribution server 2 * distributes themultiplexed data D13 to the personal computer 3, and then exits themotion picture distribution process procedure of the routine RT1 in stepSP11.

As described above, the motion picture distribution server 2, uponreceiving motion picture distribution request data Dre transmitted fromthe personal computer 3, generates the display specification informationD25 for each frame data set, specifying the display status of thescenery object 31, the telop object 33, and the person object 35 to bedistributed, and compresses and encodes the display specificationinformation D25 to generate the encoded display specification data D26,distributing it to the personal computer 3.

Furthermore, the motion picture distribution server 2 tracks thetracking areas 32, 34, and 36 in the frame pictures based on the subjectframe data D1 and extracts the objects 31, 33, and 35 from the trackingareas 32, 34, and 36, respectively, and sequentially compresses andencodes the scenery object data D5, the telop object data D8, and theperson object data D11 of the extracted objects 31, 33, and 35 togenerate encoded scenery object data D6, encoded telop data D9, andencoded person data D12, distributing them to the personal computer 3.

1.4 Construction of Personal Computer

Referring to FIG. 7, a control unit 20 of a display control unit 29 ofthe personal computer 3 has a basic program for executing basicoperations on the personal computer 3, such as Windows 2000 (trademarkof Microsoft Corporation), loaded in advance in an internal RAM, so thatwhen a predetermined operation for requesting distribution of a motionpicture is made via a mouse 21 for input operations, the control unit 20generates motion picture request data Dre and transmits it to the motionpicture distribution server 2 on the distribution side.

Upon receiving the multiplexed data D13 distributed from the motionpicture distribution server 2 via a reception sidetransmission/reception unit 24, a demultiplexing unit 25 demultiplexesthe multiplexed data D13 into the encoded display specification dataD26, the encoded scenery data D6, the encoded telop data D9, and theencoded person data D12, and forwards them to a display specificationinformation decoding unit 28, an arbitrary shape decoding unit 26A, anarbitrary shape decoding unit 26B, and an arbitrary shape decoding unit26C, respectively.

Upon receiving the encoded scenery data D6 forwarded from thedemultiplexing unit 25, the arbitrary shape decoding unit 26A decodesthe encoded scenery data D6 to restore the original scenery object dataD5, and stores it in an individual frame memory 27A of the displaycontrol unit 29.

Similarly to the arbitrary shape decoding unit 26A, the arbitrary shapedecoding units 26B and 26C, upon receiving the encoded telop data D9 andthe encoded person data D12 forwarded from the demultiplexing unit 25,decodes the encoded telop data D9 and the encoded person data D12 torestore the original telop object data D8 and the original person objectdata D11, and store them in individual frame memories 27B and 27C of thedisplay control unit 29, respectively.

The display specification information decoding unit 28, upon receivingthe encoded display specification data D26 forwarded from thedemultiplexing unit 25, decodes the encoded display specification dataD26 to restore the original display specification information D25, andforwards it to the control unit 20 of the display control unit 29.

Upon receiving the display specification information D25 forwarded fromthe display specification information decoding unit 28, the control unitwrites the display specification information D25 to the internal RAM.The layer order specification data D22 in the display specificationinformation D25 is arranged such that the control unit.20 repeatedlyreads from the individual frame memories 27C, 27B, and 27A in that orderaccording to the specified layer order (the scenery object 31 on theuppermost layer, the telop object 33 on the intermediate layer, and theperson object 35 on the lowermost layer).

Thus, the control unit 20 repeatedly reads the person object data D11,the telop object data D8, and the scenery object data D5 in that orderfrom the individual frame memories 27C, 27B, and 27A, respectively,according to the layer order specification data D22 in the displayspecification information D25, and stores them in a display frame memory22.

Then, the control unit 20 determines the configuration as to positionand size (hereinafter referred to as display configuration) of each ofthe person object 35,based on the person object data D11, the telopobject 33 based on the telop object data D8, and the scenery object 31based on the scenery object data D5 in the display frame memory 22,according to the display specification data D21, the center coordinatepoint and the reference point of each of the objects, and the picturesize of each of the objects in the display specification informationD25.

The control unit 20 outputs the person object 35, the telop object 33,and the scenery object 31 to the display unit 23 according to thedisplay configuration thus determined, whereby the objects 31, 33, and35 are displayed on the display unit 23.

More specifically, when a predetermined operation for requestingdistribution of a motion picture is made via the mouse 21, the controlunit 20 displays the motion picture on the display screen of the displayunit 23, or loads a motion picture reconfiguration program for modifyingthe display configuration of the motion picture on display into theinternal RAM, displaying a window screen 50 as shown in FIG. 8 on adesktop screen of the display unit 23 via the display frame memory 22.

The window screen 50 includes, from a predetermined end to the otherend, a title bar 51, a tool bar 52, a menu bar 53, and a display area 54for displaying various pictures, in that order.

In this example, the control unit 20 outputs the objects in the displayframe memory 22 with the same display configuration as that of the framepicture (FIG. 3) prior to the extraction at the motion picturedistribution server 2 on the distribution side, according to the displayspecification data D21, the center coordinate point and the referencepoint of each of the objects, and the picture size of each of theobjects in the display specification information D25 written to theinternal RAM, so that the scenery object 31 is displayed in theproximity of the top left corner of the display area 54 of the windowscreen 50, the telop object 33 in the proximity of the top right corner,and the person object 35 in the proximity of the bottom right corner, asshown in FIG. 9.

Furthermore, the control unit 20, upon receiving the displayspecification information D25 for each frame data set, newly forwardedfrom the display specification information decoding unit 28, updates thecontents of the display specification information D25 written to theinternal RAM with the contents of the new display specificationinformation D25.

The control unit 20 repeatedly reads the person object data D11, thetelop object data D8, and the scenery object data D5, in that order,from the individual frame memories 27C, 27B, and 27A, respectively,according to the updated display specification information D25, andstores them in the display frame memory 22. The control unit 20 thendetermines display configuration of the objects 31, 33, and 35 in thedisplay frame memory 22 and outputs the result to the display unit 23.

Upon each time updating the contents of the display specificationinformation D25 with new the display specification information D25sequentially forwarded from the display specification informationdecoding unit 28, the control unit 20 determines display configurationof the scenery object 31, the telop object 33, and the person object 35based on the scenery object data D5, the telop object data D8, and theperson object data D11 in the display frame memory 22, respectively,according to the updated display specification information D25 andoutputs the result to the display unit 23, so that the objects 31, 33,and 35 are displayed on the display area 54 of the window screen 50 onthe display unit 23 in real time at the same positions and with the sameshapes as those in the motion picture based on the motion picture datarecorded in the HDD 6 of the motion picture distribution server 2 on thedistribution side.

When, for example, the mouse 21 is pressed with a cursor 56 (FIG. 9)placed on the telop object 33, the cursor 56 is moved to the proximityof the bottom left corner with the mouse 21 pressed, and then the mouse21 is unpressed, that is, when the telop object 33 is dragged, thecontrol unit 20 moves the telop object 33 from the proximity of the topright corner of the window screen 50 to the proximity of the bottom leftcorner in accordance with the movement of the cursor 56, as shown inFIG. 10.

That is, when the mouse 21 is pressed, the control unit 20 sequentiallydetects objects associated with the relevant coordinate position fromthe uppermost layer to the lowermost layer, and determines that thetelop object 33 is specified when the telop object 33 is detected.

The control unit 20 continues changing the reference point of the telopobject 33 in the display specification information D25 in accordancewith the distance and direction of the movement of the telop object 33.At this time, the control unit 20 outputs the telop object 33 to thedisplay unit 23 while modifying the display configuration of the telopobject 33 in the display frame memory 22 using the center coordinatepoint of the telop object 33 based on the modified reference point, sothat the telop object 33 is displayed with movement in the display area54 of the window screen 50.

Thus, the control unit 20 disregards movement inherent to the telopobject 33 between adjacent frames while the telop object 33 is beingmoved by the mouse 21, and reflects the inherent movement of the telopobject 33 between adjacent frames at the position in accordance with thedistance and direction of the movement of the telop object 33 by themouse 21.

Furthermore, when the mouse 21 is unpressed, the control unit 20 fixesthe reference point to the current modified reference point so that themodified reference point will be maintained as the reference point ofthe telop object 33 for each subsequent display specificationinformation D25 forwarded from the display specification informationdecoding unit 28 for each frame data set.

Thus, even if he telop object 33 is moved to the proximity of the bottomleft corner of the window screen 50, the control unit 20 determinesdisplay configuration of the telop object 33 using the center coordinatepoint of the telop object 33 based on the modified reference point, sothat the telop object 33 is displayed in real time at the correctposition.

The menu bar 53 of the window screen 50 includes an erase item 53A forerasing an object displayed in the display area 54, a cancel erase item53B for cancelling erasure of an erased object, an enlarge/reduce item3C for enlarging or reducing an object, and a change layers item 53D forchanging order of layers between objects.

Thus, when the mouse 21 (FIG. 7) is pressed with the cursor 56 placed atan arbitrary point and is then immediately unpressed, that is, when themouse 21 is clicked, the control unit 20 becomes ready to acceptselection of one of the items 53A to 53D on the menu bar 53.

That is, subsequent to the state described above with reference to FIG.10, when the mouse 21 is clicked with the cursor 56 placed on the telopobject 33and then on the erase item 53A on the menu bar 53, decodingprocess by the arbitrary shape decoding unit 26B (FIG. 7), from whichthe telop object 33 has been forwarded, is halted to stop forwarding ofthe telop object data D8 from the arbitrary shape decoding unit 26B tothe display frame memory 22 via the individual frame memory 27B, wherebythe telop object 33 is erased from the display area 54 of the windowscreen 50, as shown in FIG. 11.

Thus, the control unit 20 erases the telop Object 33 in response to arequest from a user operating the mouse 21, and also serves to minimizethe overall power consumption of the personal computer 3.

When the telop object 33 is erased, the control unit 20 changes thedisplay specification data D21 in the display specification informationD25 written to the internal RAM to non-display setting only for thetelop object 33, so that non-display setting is maintained for the telopobject 33 in subsequent display specification information D25 forwardedfrom the display specification information decoding unit 28 for eachframe data set.

Thus, even when the telop object 33 is erased from the window screen 50,the control unit 20 maintains the status of the telop object 33.

When the cancel erase item 53B on the menu bar 53 is clicked on, thecontrol unit 20 resumes the decoding process by the arbitrary shapedecoding unit 26B (FIG. 7) and changes the display specification of thetelop object 33 in the display specification data D21 to displaysetting, and the display setting is maintained.

Furthermore, subsequent to the state described above with reference toFIG. 11, when the mouse 21 is clicked with the cursor 56 placed on thescenery object 31 and then on the enlarge/reduce item 53C, the controlunit 20, as shown in FIG. 12, prompts the user with a message saying,for example, “drag corners” and displays an outer edge EG (the edge ofthe hatched area in FIG. 4, or the edge of the scenery tracking area 32in FIG. 3), which is not displayed in the scenery object 31, in brokenline.

In this state, if, for example, the bottom right corner of the edge EG,shown in broken line, is dragged towards the bottom right of the displayarea 54 of the window screen 50, the control unit 20 enlarges thescenery object 31 in the edge EG, shown in broken line, in accordancewith the movement, as shown in FIG. 13.

That is, the control unit 20 continues modifying the picture size of thescenery object 31 in the display specification information D25 inaccordance with the distance and direction of the movement of thescenery object 31. At this time, the control unit 20 reads the sceneryobject data D5 from the individual frame memory 27A and stores it in thedisplay frame memory 22 while sequentially interpolating the sceneryobject data D5 according to the modified picture size without changingthe aspect ratio. The control unit 20 then outputs the scenery object 31based on the interpolated scenery object data D5 (FIG. 7) to the displayunit 23, displaying the enlarged scenery object 31 in the display area54 of the window screen 50.

Furthermore, when the mouse 21 is unpressed, the control unit 20 fixesthe picture size to the current modified picture size so that thepicture size of the scenery object 33 is maintained to the modifiedpicture size for each subsequent display specification information D25forwarded from the display specification information decoding unit 28for each frame data set.

Thus, when the scenery object 31 is enlarged, the control unit 20displays the scenery object 31 in real time in the enlarged size.

On the other hand, when the user is prompted with a message saying, forexample, “drag corners” in the instruction item 55 with the edge EGdisplayed in broken line, for example, if the bottom right corner of theedge EG is dragged towards the top left of the display area 54 of thewindow screen 50, the control unit 20 reduces the scenery object 31 inthe edge EG, shown in broken line, in accordance with the movement.

Furthermore, subsequent to the state described above with reference toFIG. 13, for example, if the mouse 21 is clicked with the cursor 56placed on the scenery object 31 and then on the change layers item 53Don the menu bar 53, the control unit 20, as shown in FIG. 14, promptsthe user with a message saying, for example, “select next object” anddisplays the outer edge EG, not displayed in the scenery object 31, inbroken line.

In this state, when, for example, the mouse 21 is clicked with thecursor 56 placed on the person object 35, the control unit 20 reverses,the order of layers between the scenery object 31 and the person object35, as shown in FIG. 15.

That is, with regard to the scenery object 31 on the uppermost layer,the telop object 33 on the intermediate layer, and the person object 35on the lowermost layer, specified in advance by the layer orderspecification data D22 in the display specification information D25, thecontrol unit 20 changes layers so that the scenery object 31 is now onthe lowermost layer and the person object on the uppermost layer,maintaining the modified layer order for the layer order specificationdata D22 in each subsequent display specification information D25forwarded from the display specification information decoding unit 28for each frame data set.

Thus, the control unit 20 repeatedly reads the scenery object data D5,the telop object data D8, and the person object data D11, in that order,from the individual frame memories 27A, 27B, and 27C, respectively,according to the modified layer order specification data D22, storingthem in the display frame memory 22.

The control unit 20 then determines display configuration for the personobject 35, the telop object 33, and the scenery object 31 in the displayframe memory 22 and outputs the result, whereby the person object 35 isdisplayed on the uppermost layer on the display unit 23.

As described above, the control unit 20 modifies the displayspecification information D25 according to various operations of themouse 21, controlling the individual frame memories 27A, 27B, and 27Cand the display frame memory 22 according to the modified displayspecification information D25, so that display configuration of theobjects 31, 33, and 35 on the display area 54 of the window screen 50can be modified.

1.5 Procedure of Display Status Modifying Process

The procedure of the display status modifying process will be summarizedbelow. Referring to FIG. 6, upon starting a routine RT2, the personalcomputer 3 enters step SP21.

In step SP21, the personal computer 3 transmits to the motion picturedistribution server 2 motion picture distribution request data Dregenerated in response to a predetermined operation of the mouse 21 forrequesting distribution of a motion picture, and then proceeds to stepSP22.

In step SP22, the personal computer 3 determines whether the multiplexeddata D13 distributed from the motion picture distribution server 2 hasbeen received. If the test turns out negative, the personal computer 3waits until the multiplexed data D13 is received.

On the other hand, if the test turns out positive, the personal computer3 proceeds to step SP23.

In step SP23, the personal computer 3 demultiplexes the multiplexed dataD13 into the encoded data D6, D9, and D12 (FIG. 7) and the encodeddisplay specification data D26, decodes them to restore the object dataD5, D8, and D11, and the display specification information D25,respectively, and then proceeds to step S24.

In step SP24, the personal computer 3 determines display configurationof the objects 31, 33, and 35based on the object data D5, D8, and D11,respectively, according to the display specification information D25,displays the objects 31, 33, and 35 on the display unit 23 (FIG. 9), andthen proceeds to step SP25.

In step SP25, the personal computer 3 determines whether the mouse 21has been pressed. If the test turns out negative, indicating that theuser watching the display unit 23 does not want to modify the displayconfiguration of any of the objects 31, 33, and 35, the personalcomputer 3 returns to step SP24, continuing determination of displayconfiguration according to the display specification information D25 anddisplaying the objects 31, 33, and 35 on the display unit 23.

On the other hand, if the test turns out positive, indicating that theuser watching the display unit 23 wants to modify the displayconfiguration of the objects 31, 33, and 35 and that one of the objects31, 33, and 35 has been selected, the personal computer 3 proceeds tostep SP26.

In step SP26, the personal computer 3 determines whether an objectassociated with the position at which the mouse 21 has been pressed ispresent. If the test turns out negative, indicating none of the objectsis present at the position at which the mouse 21 has been pressed, thepersonal computer 3 returns to step SP25.

On the other hand, if the test turns out positive, indicating that anobject is present at the position at which the mouse 21 has been pressedand that the object has been selected, the personal computer 3 proceedsto step SP27.

In step SP27, the personal computer 3 determines whether the mouse 21has been unpressed. If the test turns out negative, the personalcomputer 3 proceeds to step SP28 (FIG. 17).

In step SP28, the personal computer 3 determines whether the mouse 21has been moved. If the test turns out negative, the personal computer 3returns to step SP27 (FIG. 16), and waits until the mouse 21 is moved.

On the other hand, if the test turns out positive, indicating that themouse is kept pressed being moved, the personal computer 3 proceeds tostep SP29.

In step SP29, the personal computer 3 modifies the reference point ofthe selected object in the display specification information D25 inaccordance with the distance and direction of the movement of the mouse21, and then proceeds to step SP30.

In step SP30, the personal computer 3 determines whether the mouse 21has been unpressed. If the test turns out negative, the personalcomputer 3 continues changing the reference point of the selected objectin the display, specification information D25 in accordance with thedistance and direction of the movement of the mouse, 21 until the mouse21 is unpressed.

On the other hand, if the test turns out positive, the personal computer3 proceeds to step SP31.

In step SP31, the personal computer 3 fixes the reference point of theselected object in the display specification information D25 to thereference point at the time when the mouse 21 is unpressed. The personalcomputer 3 then returns to step SP24 (FIG. 16) and determines displayconfiguration of the selected object based on the fixed reference pointand using the center coordinate point of the selected object (FIG. 10).

If the test in step SP27 (FIG. 16) turns out positive, indicating thatan operation associated with one of the items 53A to 53D on the menu bar53 (FIG. 8) is to be executed on the selected object, the personalcomputer 3 proceeds to step SP32 (FIG. 17).

In step SP32, the personal computer 3 determines whether the erase item53A has been clicked on. If the test turns out positive, the personalcomputer 3 enters a subroutine SRT1 shown in FIG. 18 to execute anerasing process.

Upon starting the subroutine SRT1, the personal computer enters stepSP40.

In step SP40, the personal computer 3 halts decoding process for encodeddata associated with the object selected in step SP26 (FIG. 16), erasingthe selected object (FIG. 11), and then proceeds to step SP41.

In step SP41 the personal computer 3 modifies the display specificationdata D21 in the display specification information D25 to non-displaysetting only for the selected object. The personal computer 3 thenreturns to step SP24 (FIG. 16) and continues determination of displayconfiguration according to the modified display specificationinformation D25.

If the test in step SP32 turns out negative, indicating that erasure ofthe selected object is not desired, the personal computer 3proceeds tostep SP33.

In step SP33, the personal computer 3 determines whether the cancelerase item 53B has been clicked on. If the test turns out positive,indicating that the selected object has been erased before and thaterasure of the selected object is now desired to be cancelled, thepersonal computer 3 enters a subroutine SRT2 shown in FIG. 19 to executean erasure cancelling process.

Upon starting the subroutine SRT2, the personal computer 3 enters stepSP50.

In step SP50, the personal computer 3 resumes the decoding process forthe encoded data associated with the object selected in step SP26 (FIG.16), and then proceeds to step SP51.

In step SP51, the personal computer 3 modifies the display specificationdata D21 of the selected object in the display specification informationD25 from non-display setting to display setting. The personal computer 3then returns to step SP24 (FIG. 16) and continues determination ofdisplay configuration according to the modified display specificationinformation D25.

If the test in step SP33 turns out negative, indicating that cancellingof erasure of the selected object is not desired, the personal computer3 proceeds to step SP34.

In step SP34, the personal computer 3 determines whether theenlarge/reduce item 53C has been clicked on. If the test turns outpositive, the personal computer 3 enters a subroutine SRT3 shown in FIG.20 to execute an enlarging/reducing process.

Upon starting the subroutine SRT3, the personal computer 3 enters stepSP60.

In step SP60, the personal computer 3 determines whether the mouse 21has been pressed at one of the four corners of the edge EG (FIG. 12)including the selected object. If the test turns out negative, thepersonal computer 3 waits until the mouse 21 is pressed at one of thefour corners of the edge EG including the selected object.

On the other hand, if the test turns out positive, the personal computer3 proceeds to step SP61.

In step SP61, the personal computer 3 modifies the picture size of theselected object in the picture size of the display specificationinformation D25 in accordance with the distance and direction of themovement of the mouse 21 (FIG. 13), and then proceeds to step SP62.

In step SP62, the personal computer 3 interpolates object data of theselected object in accordance with the modified picture size, and thenproceeds to step SP63.

In step SP63, the personal computer 3 determines whether the mouse 21has been unpressed. If the test turns out negative, the personalcomputer 3 returns to step SP62 and modifies the picture size of theselected object in accordance with the distance and direction of themovement of the mouse 21.

On the other hand, if the test turns out positive, indicating that theenlarging or reducing operation for the selected object via the mouse 21has been finished, the personal computer 3 proceeds to step SP64.

In step SP64, the personal computer 3 fixes the picture size to thepicture size at the time when the mouse 21 is unpressed. The personalcomputer 3 then returns to step SP24 (FIG. 16) and continuesdetermination of display configuration according to the fixed displayspecification information D25.

If the test in step SP34 turns out negative, indicating thatenlarging/reducing of the selected object is not desired, the personalcomputer 3 proceeds to step SP35.

In step SP35, the personal computer 3 determines whether the changelayers item 53D has been clicked on. If the test turns out positive, thepersonal computer 3 enters a subroutine SRT4 shown in FIG. 21 to executea layer changing process.

Upon starting the subroutine SRT4, the personal computer 3 enters stepSP70.

In step SP70, the personal computer 3 prompts the user to select a newobject for which layers are to be changed with the selected object (FIG.14), and then proceeds to step SP71.

In step SP71, the personal computer 3 determines whether a new objectfor which layer is to be changed has been selected. If the test turnsout negative, the personal computer 3 waits until a new object isselected.

On the other hand, if the test turns out positive, the personal computer3 proceeds to step SP72.

In step SP72, the personal computer 3 changes layers in the layer orderspecification data D22 between the object selected in step SP26 (FIG.16) and the new object selected in step SP71 (FIG. 15). The personalcomputer 3 then returns to step SP24 (FIG. 16) and continuesdetermination of display configuration according to the modified displayspecification information D25.

If the test in step SP35 turns out negative, indicating that none of theitems 53A to 53D on the menu bar 53 has been selected, the personalcomputer 3 loops through steps SP32, SP33, SP34, and SP35 until one ofthe items 53A to 53D is clicked on.

As described above, the personal computer 3 sequentially decodes theencoded display specification data D26 distributed separately from theencoded data D6, D9, and D12, displays the scenery object 31, the telopobject 33, and the person object 35 based on the object data D5, D8, andD11, respectively, in a display configuration determined according tothe display specification information D25 thus obtained, and whenvarious operations are executed via the mouse 21, the personal computer3 modifies the display specification information D25 in accordance withthe operation of the mouse 21, displaying the objects 31, 33, and 35 ina display configuration determined according to the modified displayspecification information D25.

1.6 Operation and Advantages

According to the motion picture distribution system described above, themotion picture distribution server 2 tracks the tracking areas 32, 34,36 respectively by the object processing units CP1 to CP3 thatconstitute object processing means, and distributes the extractedobjects 31, 33, and 35, and the display specification information D25specifying display status of the objects 31, 33, and 35, generated foreach frame data set by the display specification information generationunit 13 that constitutes display specification information generationmeans, from the distribution side transmission/reception unit 12 thatconstitutes transmission means to the personal computer 3 via theInternet 4.

The personal computer 3 receives the objects 31, 33, and 35, and thedisplay specification information D25 via the Internet 4 by thereception side transmission/reception unit 24 that constitutes receptionmeans, and determines display configuration according to the displayspecification information D25 by the display control unit 29 thatconstitute display control means, achieving the same display status onthe display unit 23 that constitute display means as the frame pictures(FIG. 3) prior to extraction at the motion picture distribution server2.

When a modifying operation is made via the mouse 21 that constitutesinput means on the objects 31, 33, and 35 displayed in the same displaystatus as the frame pictures (FIG. 3) prior to extraction at the motionpicture distribution server 2, the personal computer 3 modifies thecenter coordinate point and the reference point of each of the objects,the picture size of each of the objects, the display specification dataD21, and the layer order specification data D22, that constitutecontents of the display specification information D25, by the controlunit 20 that constitutes specification content modification means, inaccordance with the modifying operation of the mouse 21.

Thus, according to the motion picture distribution system 1, the objects31, 33, and 35 can be provided with intention of the user operating themouse 21 reflected thereon.

In this case, the personal computer 3 uses the center coordinate pointand the reference point of each of the objects, the picture size of eachof the objects, the display specification data D21, and the layer orderspecification data D22 as the contents of display specification, so thatthe objects 31, 33, and 35 displayed on the display unit 23 can bedisplayed in different layer order, moved to different positions,enlarged or reduced, and can be erased.

Thus, the personal computer 3 is allowed to perform simple processing onthe objects 31, 33, and 35 displayed in real time without using a largescale apparatus such as an editing apparatus.

According to the motion picture distribution system described above, theobjects 31, 33, and 35, and the display specification information D25generated by the motion picture distribution server 2 are distributed tothe personal computer 3, and the personal computer 3 displays theobjects 31, 33, and 35 in display configuration determined according tothe display specification information D25 and modifies specificationcontents of the display specification information D25 in accordance withmodifying operations via the mouse 21, so that Intention of the useroperating the mouse 21 can be reflected on the objects 31, 33, and 35,serving to popularize distribution of picture.

Although the first embodiment has been described in relation to a casewhere motion picture data MD is read from the HDD 6 by the unit of eachsubject frame data D1 to detect motion vector, the present invention isnot limited thereto, and motion vector may be sequentially detected foreach subject frame data of motion picture data acquired by imaging atarget with a video camera.

Furthermore, although the first embodiment has been described inrelation to a case where the control unit 20 of the personal computer 3determines display configuration of the objects 31, 33, and 35 accordingto the display specification information D25 transmitted from the motionpicture distribution server 2, the present invention is not limitedthereto, and the control unit 20 of the personal computer 3 maydetermine display configuration according to prestored displayspecification information.

Furthermore, although the first embodiment has been described inrelation to a case where the motion picture distribution server 2multiplexes and distributes the encoded data D6, D9, and D12, and theencoded display specification data D26, the present invention is notlimited thereto, and the encoded data D6, D9, and D12, and the encodeddisplay specification data D26 may be distributed separately.

Furthermore, although the first embodiment has been described inrelation to a case where the display specification information D25 isgenerated for each frame data set of 60 frames, the present invention isnot limited thereto, and the display specification information D25 maybe generated for each frame data set of arbitrary number of frames, forexample, 20 frames or 37 frames.

Furthermore, although the first embodiment has been described inrelation to a case where the personal computer 3 is used as a displaystatus modifying apparatus, the present invention is not limitedthereto, and may also be applied to various display status modifyingapparatuses in the form of, for example, PDAs (Personal DigitalAssistants) or cellular phones.

In this case, the display status modifying apparatuses halt the decodingprocess by the arbitrary shape decoding units 26A to 26C from which theobjects 31, 33, and 35 originate, as specified by the mouse 21. Thus, byselectively erasing the objects 31, 33, and 35 in accordance withavailable processing capability, usability of the display statusmodifying apparatuses can be improved.

2. Second Embodiment

2.1 Overall Construction

Referring to FIG. 22, a motion picture providing system 100 according toa second embodiment of the present invention is constructed by linkingarbitrary shape encoding apparatuses 62A, 62B, and 62C, with personalcomputers 63 and 64 via the Internet 65.

The arbitrary shape encoding apparatus 62A tracks a tracking area inframe pictures based on frame data constituting motion picture dataacquired by imaging a target with a video camera 61A, and sequentiallycompresses and encodes each object data of an object extracted from thetracking area to generate encoded data, transmitting it to the personalcomputer 63.

Similarly to the arbitrary shape encoding apparatus 62A, the arbitraryshape encoding apparatuses 62B and 62C track tracking areas in framepictures based on frame data constituting motion picture data acquiredby imaging targets by video cameras 61B and 61C, and sequentiallycompress and encode each object data of objects extracted from thetracking areas to generate encoded data, respectively, transmitting themto the personal computer 63.

The personal computer 63 sequentially decodes the encoded datatransmitted from the arbitrary shape encoding apparatuses 62A, 62B, and62C, and displays objects based on the decoded object data in real time.Furthermore, the personal computer 63 reencodes the objects as requiredto generate encoded data, transmitting them to the personal computer 64.

2.2. Construction of Arbitrary Shape Encoding Apparatuses

The arbitrary shape encoding apparatuses 62A, 62B, and 62C areconstructed identically to each other, and thus only the construction ofthe arbitrary shape encoding apparatus 62A will be described herein.

Referring to FIG. 23, a motion vector detection unit 70 of the arbitraryshape encoding apparatus 62A, upon receiving motion picture distributionrequest data Dre1 transmitted from the personal computer 63, divideseach frame data of motion picture data MD1 supplied from the videocamera 61A into a plurality of macro blocks, generating divided framedata D30.

The motion vector detection unit 70 reads, for example, frame datatemporally preceding the divided frame data as reference frame data, anddetects motion vector data D31 of each of the macro blocks by blockmatching using the divided frame data D30 and the reference frame data,forwarding the motion vector data D31 and the divided frame data D30 toan object processing unit CP4 that constitutes object processing means.

In this embodiment, similarly to the example described earlier withreference to FIG. 3, in a motion tracking unit 71 of the objectprocessing unit CP4, a person tracking area 36 including a person object35 in the proximity of the bottom right corner of the frame picture 30based on frame data is specified in advance by tracking specificationdata D35.

Thus, the motion tracking unit 71 uses the motion vector data D31 of allthe macro blocks associated with the person tracking area 36 (FIG. 3) inthe motion vector data D31 forwarded from the motion vector detectionunit 70 to roughly detect the movement of each of the macro blocksassociated with the person tracking area 36 between each adjacent pairof divided frame data D30 forwarded from the motion vector detectionunit 70.

Furthermore, the motion tracking unit 71 detects the person trackingarea 36, having been roughly detected, now more specifically by linematching between each adjacent pair of divided frame data 30, so thatthe movement of each of the macro blocks associated with the persontracking area 36 is detected with high precision.

Furthermore, similarly to the example described earlier with referenceto FIG. 4, the motion tracking unit 71 separates the person trackingarea 36 specified by the tracking specification data D35, and forwardsit to an arbitrary shape extraction unit 72 as person tracking data D32.

The arbitrary shape extraction unit 72 is adapted to accept fromoutside, when the tracking specification data D35 is input to the motiontracking unit 71, input of extraction specification data D36 specifyingan arbitrary point on an object, whereby an object to be extracted fromthe tracking area specified by the tracking specification data D35 isspecified.

In this embodiment, in the arbitrary shape extraction unit 72, similarlyto the example described earlier with reference to FIG. 3, an arbitrarypoint 35A specifying the person object 35 is specified by the extractionspecification data D36.

Thus, similarly to the example described earlier with reference to FIG.4, the arbitrary shape extraction unit 72 detects the edge of the personobject 35 from the person tracking area 36 based on the person trackingdata D32 forwarded from the motion tracking unit 71, with an assumptionthat the center of the person object 35 is a pixel associated with thepoint 35A specified by the extraction specification data D36, leavingthe original picture inside the edge as it is while replacing theoutside with, for example, pixels representing black so that it will notbe displayed on the reception side, whereby the person object 35 isextracted.

Furthermore, similarly to the example described earlier with referenceto FIG. 5, the arbitrary shape extraction unit 72 creates a mask patternpicture 37 for exposing the extracted person object 35 while coveringthe surrounding area of the person object 35 so that the shape of theperson object 35 will be presented to the reception side.

The arbitrary shape extraction unit 72 forwards object data of theextracted person object 35 (in this case including the surrounding areain the person tracking area 36, replaced with black pixels), and maskpattern data of the mask pattern picture 37 to an arbitrary shapeencoding unit 73 as person object data D33.

The arbitrary shape encoding unit 73 compresses and encodes by the unitof macro blocks of the same as those in the original picture, the objectdata in the person object data D33 by a relatively low compression ratebased on MPEG4, generating encoded data, and also compresses and encodesthe mask pattern data in the person object data D33 by a relatively lowcompression rate, generating encoded mask pattern data.

The arbitrary shape encoding unit 73 forwards the encoded data and theencoded mask pattern data via a transmission/reception unit 74 to thepersonal computer 63 as encoded person data D34.

The arbitrary shape encoding apparatus 62B is constructed identically tothe arbitrary shape encoding apparatus 62A. In this embodiment, as shownin FIG. 24A, the arbitrary shape encoding apparatus 62B tracks theentire frame picture (hereinafter referred to as a background trackingarea) 77 based on frame data constituting motion picture data suppliedfrom the video camera 61B, extracts the entire tracking area as anobject (hereinafter referred to as a background object) 77, andsequentially compresses and encodes each data of the object 77(hereinafter referred to as background object data) D37 to generateencoded background data D38, transmitting it to the personal computer63.

The arbitrary shape encoding apparatus 62C is constructed identically tothe arbitrary shape encoding apparatus 62A. In this embodiment, as shownin FIG. 24B, the arbitrary shape encoding apparatus 62C tracks atracking area (hereinafter referred to as a flower tracking area) 79including a picture of a flower (hereinafter referred to as a flowerobject) 78 in the proximity of the bottom right corner of the framepicture based on frame data constituting: motion picture data suppliedfrom the video camera 61C, extracts the flower object 78 from the flowertracking area 79, and sequentially compresses and encodes each data ofthe flower object 78 (hereinafter referred to as flower object data) D39to generate encoded flower data D40, transmitting it to the personalcomputer 63.

2.3 Construction of Personal Computer

Referring to FIG. 25, in which parts corresponding to those in FIG. 7are numbered the same, a control unit 80 of a display control unit 29has a basic program for running basic operations on a personal computer63, such as Windows 2000 (trademark of Microsoft Corporation), loaded inan internal RAM, so that when a predetermined operation for requestingdistribution of a motion picture is made via a mouse 21 for inputoperations, the display control unit 29 generates motion picture requestdata Dre1 and transmits it to arbitrary shape encoding apparatuses 62Ato 62C (FIG. 22).

Upon receiving encoded data D34, D38, and D40 from the arbitrary shapeencoding apparatuses 62A, 62B, and 62C, a reception sidetransmission/reception unit 24 arbitrarily allocates and forwards themto arbitrary shape decoding units 26A, 26B, and 26C. In this embodiment,the encoded person data D34 is forwarded to the arbitrary shape decodingunit 26A, the encoded background data D38 to the arbitrary shapedecoding unit 26B, and the encoded flower data D40 to the arbitraryshape decoding unit 26C.

Upon receiving the encoded person data D34 forwarded from the receptionside transmission/reception unit 24, the arbitrary shape decoding unit26A decodes the encoded person data D34 to restore the original personobject data D33, and stores it in an individual frame memory 27A of thedisplay control unit 29.

Similarly to the arbitrary shape decoding unit 26A, the arbitrary shapedecoding units 26B and 26C, upon receiving the encoded background dataD38 and the encoded flower data D40 forwarded from the reception sidetransmission/reception unit 24, decode the encoded background data D38and the encoded flower data D40 to restore the original backgroundobject data D37 and the original flower object data D39, and store themin individual frame memories 27B and 27C of the display control unit 29,respectively.

Written in advance to an internal RAM of the control unit 80 is displayspecification information D50 including layer order specification dataD51 for specifying the order of displaying objects with overlap(hereinafter referred to as a layer order), display specification dataD52 for specifying whether or not to display each object, and the centerpoint and the reference point of each object based on each object dataand the picture size of each object, stored in the individual framememories 27A to 27C. The control unit 80 allows arbitrary specificationas to display status of each of the objects based on the object dataD33, D37, and D39 stored in the individual frame memories 27A,.27B, and27C according to the display specification information D50 written inadvance.

The layer order specification data D51 allows the control unit 80 torepeatedly read object data from the individual frame memories 27A, 27B,and 27C in order according to the specified layer order.

Thus, the control unit 80 repeatedly reads the person object data D33,the background object data D37, and the flower object data D39, in thatorder, from the individual frame memories 27A, 27B, and 27C,respectively, according to the layer order specification data D51 in thedisplay specification information D50, and stores them in the displayframe memory 22.

Then, the control unit 80 determines display configuration of the personobject 35 based on the person object data D32, the background object 77based on the background object data D37, and the flower object 78 basedon the flower object data D39 in the display frame memory 22, accordingto the center coordinate point and reference point of each of theobjects and the picture size of each of the objects in the displayspecification information D50.

The control unit 80 outputs the person object 35, the background object77, and the flower object 78 to the display unit 23in the displayconfiguration determined according to the display specification data D52in the display specification information D50, displaying the objects 35,77, and 78 on the display unit 23.

More specifically, when a predetermined operation for requestingdistribution of a motion picture is made via the mouse 21, the controlunit 80 loads in the internal RAM a motion picture reconfigurationprogram for displaying a motion picture on a display screen of thedisplay unit 23 and modifying the display configuration of the motionpicture in display so that a window screen 90 as shown in FIG. 26, inwhich parts corresponding to those in FIG. 7 are numbered the same, willbe displayed on a desktop screen of the display unit 23 via the displayframe memory 22.

Thus, the control unit 80 determines display configuration in thedisplay frame memory 22 according to the display specification data D52,the center coordinate point and the reference point of each of theobjects, and the picture size of each of the objects in the displayspecification information D50 written in advance to the internal RAM andoutputs the result to the display unit 23, so that, for example, theperson object 35 is displayed in the display area 54 of the windowscreen 90 at substantially the same position and in substantially thesame size as in the frame pictures (FIG. 3) captured by the video camera61A (FIG. 22), the background object 77is displayed in the display area54 of the window screen 90 at a different position and in a differentsize from the frame pictures, (FIG. 24A) captured by the video camera61B, and the flower object 78 is displayed in the display area 54 of thewindow screen 90 at a different position and in a different size fromthe frame pictures (FIG. 24B) captured by the video camera 61C.

As described above, the control unit 89 determines display configurationof the objects 35, 77, and 78 based on the object data D33, 37, and 39,distributed from the arbitrary shape encoding apparatuses 62A to 62C inresponse to transmission of motion picture request data Dre1 and thendecoded, respectively, in the display frame memory 22 according to thedisplay specification information D50 written in advance in the internalRAM.

Then, the control unit 80 sequentially outputs the objects 35, 77, and78 according to the display configuration to the display unit 23,displaying the objects 35, 77, and 78 in real time in the display area54 of the window screen 90 on the display unit 23.

Similarly to the example described earlier with reference to FIG. 10,for example, when the mouse 21 is pressed with the cursor 56 placed onthe background object 77, the cursor 56 is moved to the bottom right ofthe window screen 90 with the mouse 21 kept pressed, and then the mouse21 is unpressed, that is, when a drag operation is performed, thecontrol unit 80 moves the background object 77 from the center of thewindow screen 90 to the bottom right in accordance with the movement ofthe cursor 56, as shown in FIG. 28.

That is, the control unit 80 continues changing the reference point ofthe background object 77 in the preset display specification informationD50 in accordance with the distance and direction of the movement of thebackground object 77. At this time, the control unit 80 outputs thebackground object 77 to the display unit 23 while modifying displayconfiguration of the background object 77 in the display frame memory 22using the center coordinate point of the background object 77 based onthe modified reference point, whereby the background object 77 isdisplayed in the display area 54 of the window screen 90 with movement.

Thus, the control unit 80 disregards movement inherent to the backgroundobject 77 between adjacent frames while, the background object 77 isbeing moved by the mouse 21, and reflects the inherent movement of thebackground object 77 between adjacent frames at the position inaccordance with the distance and direction of the movement of thebackground object 77 by the mouse 21.

Furthermore, when the mouse 21 is unpressed, the control unit 80 fixesthe reference point to the current modified reference point. Thus, evenif the background object 77 is moved to the bottom right of the windowscreen 90, the control unit 80 displays the background object 77 at thecorrect position in real time.

Furthermore, subsequent to the state described above with reference toFIG. 28, similarly to the example, described earlier with reference toFIG. 11, for example, when the mouse 21 is clicked with the cursor 56placed on the flower object 78 and then on the erase item 53A on themenu bar 53, decoding process by the arbitrary shape decoding unit 26C(FIG. 24), from which the flower object 77 has been forwarded, is haltedto stop forwarding of the flower object data D39 from the arbitraryshape decoding unit 26C to the display frame memory 22 via theindividual frame memory 27C, whereby the flower object 78 is erased fromthe display area 54 of the window screen 90, as shown in FIG. 29.

Thus, the control unit 20 erases the flower object 78 in response to arequest from a user operating the mouse 21, and also serves to minimizethe overall power consumption of the personal computer 63.

In this case, the control unit 80 changes the display specification dataD52 in the display specification information D50, written to theinternal RAM, to non-display setting only for the flower object 78.Thus, even when the flower object 78 is erased from the window screen90, the control unit 80 maintains the status of the flower object 78.

When the cancel erase item 53B on the menu bar 53 is clicked on, thecontrol unit 80 resumes the decoding process by the arbitrary shapedecoding unit 26C (FIG. 25) and updates the display specification of theflower object 78 in the display specification data D52 to displaysetting.

Furthermore, subsequent to the state described above with reference toFIG. 29, similarly to the example described earlier with reference toFIGS. 12 and 13, when the mouse 21 is clicked with the cursor 56 placedon the background object 77 and then on the enlarge/reduce item 53C, thecontrol unit 80 prompts the user with a message saying, for example,“drag corners” in the instruction item 55 and displays an outer edge,which is not displayed in the background object 77, in broken line.

In this state, if, for example, the top left corner of the edge, shownin broken line, is dragged towards the top left of the display area 54of the window screen 90, the control unit 80 enlarges the backgroundobject 77 in the edge, shown in broken line, in accordance with themovement, as shown in FIG. 30.

That is, the control unit 80 continues modifying the picture size of thebackground object 77 in the display specification information D50 inaccordance with the distance and direction of the movement of thebackground object 77. At this time, the control unit 80 sequentiallyreads the background object data D37 (FIG. 25) from the individual framememory 27B and stores it in the display frame memory 22 whileinterpolating the background object data D37 according to the modifiedpicture size without changing the aspect ratio. The control unit 80 thenoutputs the background object 77 based on the interpolated backgroundobject data D37 to the display unit 23, displaying the enlargedbackground object 77 in the display area 54 of the window screen 90.

Furthermore, when the mouse 21 is unpressed, the control unit 80 fixesthe picture size to the current modified picture size. Thus, even if thebackground object 77 is enlarged, the control unit 80 displays thebackground object 77 in the enlarged size in real time.

On the other hand, when the user is prompted with a message saying, forexample, “drag corners” in the instruction item 55 with the edgedisplayed in broken line, for example, if the top left corner of theedge is dragged towards the bottom right of the display area 54 of thewindow screen 90, the control unit 80 reduces the background object 77in accordance with the movement.

Furthermore, subsequent to the state described above with reference toFIG. 30, similarly to the example described earlier with reference toFIG. 14, if the mouse 21 is clicked with the cursor 56 placed on thebackground object 77 and then on the change layers item 53D on the menubar 53, the control unit 80 prompts the user with a message saying, forexample, “select next object” in the instruction item 55 and displaysthe outer edge, not displayed in the background object 77, in brokenline.

In this state, when, for example, the mouse 21 is clicked with thecursor 56 placed on the person object 35, the control unit 80 reversesthe order of layers between the background object 77 and the personobject 35, as shown in FIG. 31.

That is, the control unit 80 modifies the layer order specification dataD51 in the display specification information D50 in which the flowerobject 78 is specified to be on the uppermost layer, the backgroundobject 77 on the intermediate layer, the person object 35 on thelowermost layer so that the background object 77 is now on the lowermostlayer and the person object 35 on the intermediate layer, maintainingthe modified layer order.

Thus, the control unit 80 repeatedly reads the background object dataD37 and the person object data D33, in that order, from the individualframe memories 27A to 27C according to the modified layer orderspecification data D51, storing them in the display frame memory 22. Thecontrol unit 80 then determines display configuration of the personobject 35 and the background object 77 in the display frame memory 22and outputs the result to the display unit 23, whereby the person object35 is displayed on the intermediate layer on the display unit 23.

As described above, the control unit 80 modifies the displayspecification information D50 according to various operations of themouse 21, controlling the individual frame memories 27A, 27B, and 27Cand the display frame memory 22 according to the modified displayspecification information D50, so that display configuration of theobjects 77, 78, and 35 on the display area 54 of the window screen 90can be modified.

In addition to the items 53A to 53D, on the menu bar 53 of the windowscreen 90 (FIG. 26), an insert motion picture item 53E for inserting anew motion picture in the display area 54 is provided.

Thus, subsequent to the state described above with reference to FIG. 31,for example, if the mouse 21 is clicked with the cursor 56 placed on theinsert motion picture item 53E, the control unit 80 starts an objectextraction process by an arbitrary shape extraction, processing unit 92(FIG. 25).

In this case, similarly to the motion vector detection unit 70, themotion tracking unit 71, and the arbitrary shape extraction unit 72described earlier with reference to FIG. 23, the arbitrary shapeextraction processing unit 92 tracks a tracking area (hereinafterreferred to as an eye tracking area) 97 specified in advance in framepictures based on frame data constituting motion picture data MD5acquired by imaging a target with a video camera 91, and extracts anobject (hereinafter referred to as an eye object) 98 based on anarbitrary point on the object specified in advance, storing each objectdata 45 of the eye object 98 sequentially in an individual frame memory93.

In the internal RAM of the control unit 80, the center point, thereference point, and the picture size of the object based on the objectdata D45 stored in the individual frame memory 93 is also written asadditional display specification information D50A, so that the controlunit 80 is allowed to arbitrarily specify display status of the eyeobject 98 based on the object data 45 stored in the individual framememory 93, according to the additional display specification information50A.

Thus, the control unit 80 determines display configuration of theobjects 35, 77, and 78 based on the object data D33, D37, and D39 storedin the individual frame memories 27A, 27B, and 27C, respectively,according to the display specification information D50 written inadvance, and also determines display configuration of the eye object 98based on the object data 45 stored in the individual frame memory 93according to the additional display specification information D50A, andoutputs the results to the display unit 23, so that, for example, theobjects 35, 77, and 78, and the eye object 98 extracted by the arbitraryshape extraction processing unit 92 are displayed on the display unit23.

Furthermore, on the menu bar 53 of the window screen 90 (FIG. 26),provided in addition to the items 53A to 53D is a transmit motionpicture item 53F for transmitting a motion picture to another personalcomputer in real time after display configuration currently beingreceived has been modified.

Thus, subsequent to the state described above with reference to FIG. 33,for example, if the mouse 21 is clicked with the cursor 56 placed on thetransmit motion picture item 53F, upon each time reading the object dataD33, D37, and D45 from the individual frame memories 27A, 27B, and 93according to the display specification information D50 and theadditional display specification information D50A, the control unit 80displays the objects 35, 77, and 98 on the display unit 23 via thedisplay frame memory 22 (FIG. 33) and also forwards the object data D33,D37, and D45 to a transmission processing unit 94.

At this time, the control unit 80 also forwards to the transmissionprocessing unit 94 the display specification information D50 and theadditional display specification information D50A at the time when thetransmit motion picture item 53F is clicked on, and starts an encodingprocess by the transmission processing unit 94.

In this case, the transmission processing unit 94 again compresses andencodes the object data D33 and D37 forwarded from the control unit 80to generate encoded data D34 and D38, compresses and encodes the eyeobject data D45 forwarded from the control unit 80 to generate encodedeye data D53, and also compresses and encodes the display specificationinformation D50 and the additional display specification informationD50A to generate encoded display specification data D54 and encodedadditional display specification information D54A. The transmissionprocessing unit 94 then multiplexes the encoded data D34, D38, and D53,and the encoded display specification data D54 and the encodedadditional display specification data D54A to generate multiplexed dataD55, transmitting it to the personal computer 64.

As described above, the control unit 80 controls the individual framememories 27A, 27B, and 27C, and the display frame memory 22 according tothe modified display specification information D50 so that a new objectmay be added as required to the display area 54 of the window screen 90in which the objects are displayed, and so that data of the objects maybe transmitted to the personal computer 64 in real time.

The personal computer 64 is constructed identically to the personalcomputer 63, and description thereof will be omitted.

2.4 Procedure of Display Status Modifying Process

The procedure of the display status modifying process by the personalcomputer 63 (or the personal computer 64) will be summarized below.Referring to FIG. 34, in which parts corresponding to those in FIG. 16are numbered the same, upon starting a routine RT3, the personalcomputer 63 (64) enters step SP21.

In step SP21, the personal computer 63 (64) transmits to the arbitraryshape encoding apparatuses 62A to 62C motion picture distributionrequest data Dre1 generated in response to a predetermined operation ofthe mouse 21 for requesting distribution of a motion picture. Then instep SP22, the personal computer 63 (64) receives the encoded data D34,D38, and D40 (FIG. 25) distributed from the arbitrary shape encodingapparatuses 62A to 62C, respectively, and then proceeds to step SP23.

In step SP23, the personal computer 63 (64) decodes the encoded dataD34, D38, and D40 to restore the object data D33, D37, and D38. Then instep SP24, the personal computer 63 (64) determines displayconfiguration of the objects 35, 78, and 79 based on the object dataD33, D37, and D38 according to the display specification information D50stored in advance in the internal RAM, displays the objects 35, 78, and79 on the display unit 23 (FIG. 27), and then proceeds to step SP25.

If the mouse 21 is pressed in step SP25 and an object is present at theposition at which the mouse 21 is pressed in step SP26, the personalcomputer 63 (64) determines in step SP27 whether the mouse 21 has beenunpressed. If the test turns out negative, the personal computer 63 (64)proceeds to step SP28 (FIG. 35).

If the mouse 21 is moved in step SP28, then in step. SP29, the personalcomputer 63 (64) modifies the reference point of the selected object inthe display specification information D50 in accordance with thedistance and direction of the movement of the mouse 21, and thenproceeds to step SP30.

If the mouse 21 is unpressed in step SP30, then in step SP31, thepersonal computer 63 (64) fixes the reference point of the selectedobject in the display specification information D50 to the referencepoint at the time when the mouse 21 is unpressed. The personal computer63 (64) then returns to step SP24 and determines display configurationof the selected object based on the fixed reference point (FIG. 28).

If the test in step SP27 turns out positive, indicating that anoperation associated with one of the items 53A to 53D on the menu bar 53(FIG. 26) is to be executed on the selected object, the personalcomputer 63 (64) proceeds to step SP32.

If the erase item 53A is clicked on in step SP32, the personal computer63 (64) executes the erasing process (FIG. 18) described earlier,halting decoding process for encoded data associated with the objectselected in step SP26 (FIG. 34) to erase the selected object (FIG. 29),and modifies the display specification data D21 in the displayspecification information D50 to non-display setting only for theselected object. The personal computer 63 (64) then returns to step SP24(FIG. 34) and continues determination of display configuration accordingto the modified display specification information D50.

If the erase item 53A is not clicked on in step SP32, indicating thaterasure of the selected object is not desired, the personal computer 63(64) proceeds to step SP33.

If the cancel erase item 53B is clicked on in step SP33, the personalcomputer 63 (64,) executes the erasure cancelling process (FIG. 19)described earlier, resuming the decoding process for the encoded dataassociated with the object selected in step SP26 (FIG. 34) and modifyingthe display specification of the selected object from non, displaysetting to display setting. The personal computer 63 (64) then returnsto step SP24 (FIG. 34) and continues determination of displayconfiguration according to the modified display specificationinformation D50.

If the cancel erase item 53B is not clicked on in step SP33, indicatingthat cancelling of erasure of the selected object is not desired, thepersonal computer 63 (64) proceeds to step SP34.

If the enlarge/reduce item 53C is clicked on in step SP34, the personalcomputer 63 (64) executes the enlarging/reducing process (FIG. 20)described earlier to modify the picture size of the selected object inthe display specification information D50 in accordance with thedistance and direction of movement of the mouse 21 (FIG. 30),interpolating object data of the selected object in accordance with themodified picture size.

When the mouse 21 is unpressed, the personal computer 63 (64) fixes thepicture size to the picture size at the time when the mouse 21 isunpressed. The personal computer 63 then returns to step SP24 (FIG. 34)and continues determination of display configuration according to thefixed display specification information.

On the other hand, if the enlarge/reduce item 53C is not clicked on instep SP34, indicating that enlarging/reducing of the selected object isnot desired, the personal computer 63 (64) proceeds to step SP35.

When the change layers item 53D is clicked on in step SP35, the personalcomputer 63 (64) executes the layer changing process (FIG. 21) describedearlier, so that when a new object for which layers are to be changedwith the selected object is selected in response to a prompt ofselection, the personal computer 63 (64) changes layers between theobject selected in step SP26 (FIG. 34) and the new object selected instep SP71 (FIG. 31). The personal computer 63 (64) then returns to stepSP24 (FIG. 34) and continues determination of display configurationaccording to the modified display specification information D50.

On the other hand, if the change layers item 53D is not clicked on instep SP35, indicating that layer is not desired to be changed for theselected object, the personal computer 63 (64) proceeds to step SP80.

In step SP80, the personal computer 63 (64) determines whether theinsert motion picture item 53E has been clicked on. If the test turnsout positive, the personal computer 63 (64) proceeds to step SP81.

In step SP81, the personal computer 63 (64) tracks a tracking area 97(FIG. 32) specified in advance in frame pictures based on frame dataconstituting the motion picture MD5 acquired by the video camera 91, andsequentially extracts an object from the tracking area 97 based on anpoint on the object arbitrarily specified in advance. The personalcomputer 63 (64) then returns to step SP24 (FIG. 34) and continuesdetermination of display configuration also for the extracted objectaccording to the display specification information D50 and theadditional display specification information D50A (FIG. 33).

On the other hand, if the test in step SP80 turns out negative,indicating that insertion of a motion picture into the selected objectis not desired, the personal computer 63 (64) proceeds to step SP82.

In step SP82, the personal computer 63 (64) determines whether thetransmit motion picture item 53F has been clicked on. If the test turnsout positive, the personal computer 63 (64) proceeds to step SP83.

In step SP83, the personal computer 63 (64) transmits to a destinationpersonal computer the encoded display specification data D54 (encodedadditional display specification data D54A) obtained by compressing andencoding the display specification information D50 (the additionaldisplay specification information D50A), and if the mouse 21 is clickedin step SP82, also transmits encoded data obtained by sequentiallycompressing and encoding objects whose display configuration is to bedetermined in step SP24 (FIG. 34) after the click operation. Thepersonal computer 63 (64) then returns to step SP24 (FIG. 34) andcontinues determination of display configuration for the objectsaccording to the display specification information D50.

On the other hand, if the test instep SP83 turns out negative,indicating that none of the items 53A to 53F on the menu bar 53 has notbeen selected yet for the selected object, the personal computer 63 (64)loops through steps SP32, SP33, SP34, SP35, SP80, and SP82 until one ofthe items 53A to 53F is clicked on.

As described above, the personal computer 63 (64) receives the encodeddata D34, D38, and D40 (FIG. 25) distributed from the arbitrary shapeencoding apparatuses 62A to 62C (the personal computer 63), determinesdisplay configuration of the objects 35, 77, 78, and 98 based on theobject data according to the display specification information D50 andoutputs the result to the display unit 23, displaying the objects inreal time on the display unit 23.

Furthermore, the personal computer 63 (64) modifies the displayspecification information D50 in accordance with various operations ofthe mouse 21, and controls the individual frame memories 27A, 27B, 27C,and 93, and the display frame memory 22 according to the modifieddisplay specification information D50, so that display configuration ofthe objects 35, 77, 78, and 98 displayed in the display area 54 of thewindow screen 90 can be modified.

In addition, the personal computer 63 (64) controls the individual framememories 27A, 27B, and 27C, and the display frame memory 22 so that anew object may be added to the display area 54 as required whiledisplaying the objects in the display area 54 of the window screen 90,and so that data of the objects can be transmitted to another personalcomputer in real time.

2.5 Operation and Advantages

According to the motion picture distribution system 100 described above,when the insert motion picture item 53E is clicked on, the personalcomputer 63 (64) generates the eye object 98 for additional display onthe display unit 23 by the arbitrary shape extraction processing unit 92that constitutes additional object generation means, generates theadditional display specification information D50A specifying displaystatus of the eye object 98 by the control unit 80 that constitutesadditional information generation means, and additionally displays theeye object 98 on the display unit 23 according to the additional displayspecification information D50A by the display control unit 29.Accordingly, intention of the user operating the mouse 21 can be furtherreflected. When the transmit motion picture item 53F is clicked on, thepersonal computer 63 (64) transmits, by the transmission processing unit94 that constitutes transmission means, the objects 35, 77, and 98displayed on the display unit 23 and the display specificationinformation D50 (the additional display specification information D50A)whose specification contents have been modified. Accordingly, an objecton which intention of the user operating the mouse 21 is reflected canbe provided to other users.

According to the motion picture distribution system 100 described above,a new object 98 can be added as required to the display area 54 and canalso be transmitted to another personal computer 64 in real time.Accordingly, in addition to the advantages of the first embodiment,intention of the user operating the mouse 21 can be further reflected onan object. Furthermore, the object with the user s intention reflectedthereon can be provided to other users, which serves to popularizedistribution of picture.

Although the second embodiment has been described in relation to a casewhere the arbitrary shape encoding apparatuses 62A to 62C and thepersonal computer 63 detects motion vector by sequentially reading eachsubject frame data D1 of motion picture data acquired by imaging targetswith the respectively associated video cameras 61A to 61C and 91, thepresent invention is not limited thereto, and motion vector may bedetected by sequentially reading each subject frame data of motionpicture data from an HDD.

Furthermore, although the second embodiment has been described inrelation to a case where the control unit 80 of the personal computer 63determines display configuration of the objects 35, 77, 78, and 98according to the display specification information D50 (the additionaldisplay specification information D50A) stored in advance, the presentinvention is not limited thereto. For example, the arbitrary shapeencoding apparatuses 62A to 62C may be each provided with the displayspecification information generation unit 13 described earlier withreference to FIG. 2 so that display specification information generatedby the display specification information generation unit 13 istransmitted from the arbitrary shape encoding apparatuses 62A to 62C anddisplay configuration of the objects is determined according to thedisplay specification information.

Furthermore, although the second embodiment has been described inrelation to a case where the personal computers 63 (64) is used as adisplay status modifying apparatus, the present invention is not limitedthereto, and may also be applied to various display status modifyingapparatuses in the form of, for example, PDAs (Personal DigitalAssistants) or cellular phones.

In this case, the display status modifying apparatuses halt the decodingprocess by the arbitrary shape decoding units 26A to 26C from which theobjects 35, 77, and 78 originate, as specified by the mouse 21. Thus, byselectively erasing the objects 35, 77, and 78 in accordance withavailable processing capability, usability of the display statusmodifying apparatuses can be improved.

3. Modifications

Although the embodiments have been described in relation to cases wheremotion vector is sequentially detected for each subject frame data D1 ofmotion picture data acquired by the video cameras 61A to 61C and 91 orrecorded in the HDD 6, the present invention is not limited thereto. Forexample, motion vector may be sequentially detected for each subjectframe data D1 of data obtained by encoding objects (the encoded scenerydata D6, the encoded telop data D9, the encoded person data D12 and D34,the encoded background data D38, and the encoded flower data D40) andrecorded on a recording medium such as a DVD (Digital Versatile Disk)externally connected.

Furthermore, although the embodiments have been described in relation tocases where the objects 31, 33, 35, 77, 78, and 98 are extracted alongthe edges thereof from the tracking areas 32, 34, 36, 77, 79, and 97,respectively, the present invention is not limited thereto. For example,as shown in FIG. 36, the arrangement may be such that an elliptic frame101 is set in advance so that extraction is performed along the frame101, or frames of various shapes, for example, star shaped on circular,may be set to the motion vector detection unit 7, allowing extraction ofarbitrary shapes.

Furthermore, although the embodiments have been described in relation tocases where the motion vector detection units 7 and 70 that detectsmotion vector by the unit of macro block are used as motion vectordetection means for detecting motion vector between frames of motionpicture data, the present invention is not limited thereto, and othertypes of motion vector detection means, for example, motion detectionmeans that detects motion vector based on optical flow, may be used aslong as motion vector between frames of motion picture data can bedetected

Furthermore, although the embodiments have been described in relation tocases where the objects 31, 33, 35, 77, 78, and 98 are extracted bytracking the tracking areas 32, 34, 36, 77, 79, and 97, respectively,for each subject frame data D1, the present invention is not limitedthereto, and the objects may be extracted when the tracking areas aretracked between an arbitrary pair of subject frame data D1. In thatcase, load of the extraction process is reduced.

Furthermore, although the embodiments have been described in relation tocases where the arbitrary shape encoding units 10A to 10C, 73, and thedisplay specification information encoding unit 14 performs compressionencoding according to MPEG4, the present invention is not limitedthereto, and other compression encoding methods for example, MPEG2 orMPEG1, may be used.

Furthermore, although the embodiments have been described in relation tocases where display status of the objects 31, 33, 35, 77, 78, and 98 ismodified via the mouse 21 that constitutes input means, the presentinvention is not limited thereto, and other types of input means, forexample, a touch panel, may be used.

Furthermore, although the embodiments have been described in relation tocases where the display control unit 29 that constitutes display statusmodification means allows erasing operation, erasure cancellingoperation, enlarging/reducing operation, and layer changing operation onobjects, the present invention is not limited thereto, and various otheroperations for modifying display status, for example, rotation, may beprovided.

Furthermore, although the embodiments have been described in relation tothe motion picture distribution system 1 constructed by linking themotion picture distribution server 2 that constitutes a motion picturedistribution apparatus and the personal computer 3 that constitutes adisplay status modifying apparatus with each other via the Internet 4,and the motion picture distribution system constructed by linking thearbitrary shape encoding apparatuses 62A to 62C that constitute motionpicture distribution apparatuses and the personal computers 63 and 64that constitute display status modifying apparatuses with each other viathe Internet 65, the present invention is not limited thereto. Thepresent invention may be applied to motion picture distribution systemsconstructed of various other combinations of motion picture distributionapparatuses, display status modifying apparatuses, and networks, forexample, a motion picture distribution system constructed by linking apersonal computer with an integrated video camera, and cellular phoneswith each other via the Internet, a motion picture distribution systemconstructed by linking a cellular phone to which a video camera isconnected, and PDAs (Personal Digital Assistants) with each other via aprivate line, or a motion picture distribution system constructed bylinking a television broadcasting apparatus and television apparatusesvia satellite.

In that case, the motion picture distribution apparatus may execute themotion picture distribution process either in hardware using the circuitblock described with reference to FIG. 2 or FIG. 23, or in software byinstalling a motion picture distribution process program for executingthe motion picture distribution process described with reference to FIG.6 on the motion picture distribution apparatus so that the motionpicture distribution apparatus executes the motion picture distributionprocess.

Furthermore, the display status modifying apparatus may execute thedisplay status modifying process either in hardware using the circuitblocks described with reference to FIGS. 7 and 25, or in software byinstalling a display status modifying program for executing the displaystatus modifying process described with reference to FIGS. 16 and 17 orFIGS. 34 and 35 on the display status modifying apparatus so that thedisplay status modifying apparatus executes the display status modifyingprocess.

Furthermore, when installing the motion picture distribution program andthe display status modifying program on the motion picture distributionapparatus and the display status modifying apparatus, respectively, aprogram storage medium storing both or one of the motion picturedistribution program and the display status modifying program may beused. Alternatively, wired or wireless communication channel, forexample, a local area network, the Internet, or digital satellitebroadcasting, may be used, and installation may be performed via variouscommunication interfaces such as routers and modems.

The program storage medium, storing both or one of the motion picturedistribution program and the display status modifying program forinstallation on the motion picture distribution apparatus and thedisplay status modifying apparatus to allow execution of the motionpicture distribution program and the display status modifying program,may be, for example, a package medium such as a floppy disk, a CD-ROM(Compact Disk Read-Only Memory), or a DVD (Digital Versatile Disk), andmay also be a semiconductor memory, a magnetic disk, on which theprograms are temporarily or permanently stored. The programs may bestored in the program storage medium via a local area network or theInternet, or may be stored via various communication interfaces such asrouters and modems.

1. A display status modifying apparatus comprising: reception means forreceiving from outside an object including a temporally continuouspicture of an arbitrary shape; display control means for displaying theobject on predetermined display means according to display specificationinformation that specifies a display status of the object; input meansfor inputting a modification instruction for modifying the displaystatus of the object displayed on said display means; and specificationcontent modification means for modifying specified content of thedisplay specification information according to the modificationinstruction. 2-11. (canceled)